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Abstract. The cosine transforms of functions on the unit sphere play an 
important role in convex geometry, the Banach space theory, stochastic ge- 
ometry and other areas. Their higher-rank generalization to Grassmann man- 
ifolds represents an interesting mathematical object useful for applications. 
We introduce more general integral transforms that reveal distinctive features 
of higher rank objects in full generality. We call these new transforms the 
composite cosine transforms, by taking into account that their kernels agree 
with the composite power function of the cone of positive definite symmetric 
matrices. We show that injectivity of the composite cosine transforms can be 
studied using standard tools of the Fourier analysis on matrix spaces. In the 
framework of this approach, we introduce associated generalized zeta integrals 
and give new simple proofs to the relevant functional relations. Our technique 
is based on application of the higher-rank Radon transform on matrix spaces. 



1. Introduction 

Let S 1 ™ -1 be the unit sphere in R n , u ■ v the usual inner product of vectors 
B,«6 S n . The classical cosine transform 

(1.1) (Tf)(u)= J f(v)\u-v\dv, ueS n -\ 

S n-1 

and its generalization 



(1.2) (T A f)(u)= J f(v)\u-v\ x dv, 

are commonly in use in convex geometry, the Banach space theory, harmonic anal- 
ysis, and many other areas; see \iis\ . |GH1| . |Ko| . |Schn| . Basic properties of T x 
(injectivity, boundedness in function spaces, and others) can be derived from the 
Funk-Hecke formula 

(1.3) T x P k = cft k {X)P k , 
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where (x) is the restriction of a homogeneous harmonic polynomial of even degree 

k prnj . 

In the last two decades a considerable attention was attracted to higher-rank 
generalizations of T and T A for functions on the Grassmann manifold G n ^ m of m- 
dimcnsional linear subspaces of R". We recall that, if 77 G G, vm , £ G G n ,i, I > 
to, and [rj\£] is the m-dimensional volume of the parallelepiped spanned by the 
orthogonal projection of a generic orthonormal frame in 77 onto £, then, by definition, 

(1.5) cr A /xo = / mmfdv 

G„, m 

(we adopt the same notation T x as in (|1.2[0 . For I > rn, the operator (|1.5|l repre- 
sents the composition of the similar one over G n ^ rn and the corresponding Radon 
transform acting from G„ jm to G n ^i PP, |GR| . Thus injectivity of T A can be studied 
using known results for the Radon transform (see |GR| and references therein) and 
the case I = m in i|1.5[l . Owing to this, in the following we assume that I = to, 
because just this case bears the basic features of the operator family l|1.5|) . 

The investigation of operators (|1.5() for A = 1 was initiated in stochastic geometry 
by Matheron Mai 1 p. 189], (see also |Mat2| ). who conjectured that the higher-rank 
cosine transform T 1 is injective as well as its rank-one prototype p. 1(1 . Matheron's 
conjecture was disproved in the remarkable paper by Goodey and Howard |GH1 . 
The higher-rank cosine transforms arise in convex geometry in the context of the 
generalized Shcphard problem for lower dimensional projections |GZ| . More general 
operators T x for A = 0,1,2,... were studied in GH2 p. 117], where, by using 
reduction to G4 2, it was proved that T A is non-injective for such A. The range of 
the A-cosine transform was characterized by Alesker and Bernstein |AB| (A = 1) 
and by Alesker pQ (any complex A), in terms of representations of the special 
orthogonal group SO{n). 

In this article we suggest a new approach to operators T x . This can be regarded 
as a complement to the well-known group representation method. The latter has 
proved to be useful in the study of Radon and cosine transforms, invariant dif- 
ferential and integro-differential operators on diverse homogeneous spaces of the 
orthogonal group; see |Goo| . |Gr| . |GH| . [Strlj . |Str2| . |TT| . and references therein. 
Our method differs from those in the cited papers. It gives a direct analog of the 
multiplier equality l|1.3l) and is applicable to a much more general operator family of 
the so-called composite cosine transforms. These are introduced in Section 3 which 
describes main results of the paper. In Section 4 we introduce the so-called gen- 
eralized zeta integrals with additional "angle component" / on the relevant Stiefel 
manifold. An important by-product of our investigation is a functional equation for 
these integrals that gives rise to the composite cosine transform T x f, for A G C m , 
in the most general form. The case / = 1 was studied in |FK| in the context 
of Jordan algebras. The argument from jFKj was extended in (Ol when / is a 
determinantally homogeneous harmonic polynomial. An alternative proof of this 
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result for zeta integrals on matrix spaces was given in |OR2| . This proof employs 
an idea from |Kh| to derive the result for A G C as a diagonal case of the more 
general statement for vector- valued A = (Ai,...A m ) G C m . This idea allows us 
to avoid essential technical difficulties (e.g., implementation of Bessel functions of 
matrix argument) which arise when we get stuck on the complex analysis of a single 
variable; cf. [EE], EH], [TTu4j . 

In the present paper, we use the same idea and suggest a new method that 
demonstrates application of the higher-rank Radon transform on matrix spaces 
|ORlj . |OR3| . This Radon transform enables us to reduce the problem to the 
known case / = 1. For the usual cosine transform on the unit sphere, this approach 
is due to A. Koldobsky jKoj . Section 5 contains proofs of the main results. 

One should note that the Fourier analysis of homogeneous distributions is one 
of the oldest topics in the theo ry of distributions, and there is a vast literature on 
this subject; see, e.g., |Es | . |GS) . |Lej . |Raj . [5a]. [5e|. 

Acknowledgement. We are thankful to Professors S. Alesker, P. Goodey and 
W. Weil for very helpful comments and pleasant discussion. 

2. Preliminaries 

The main references for this section are |FK| . |Gi) . [T] . 

2.1. Notation. Let R nxm be the space of real matrices x = (xij) having n rows 
and m columns; dx — Yi7=i Ilj=i dxij. In the following, x' denotes the transpose 
of x, I m is the identity m x m matrix, and 0(n) is the group of real orthogonal 
n x n matrices. For n > m, we denote by V n , m = {v G E nXm : v'v — I m } the 
Stiefel manifold of orthonormal m-frames in W 1 . This is a homogeneous space with 
respect to the action V n ^ m 3 v — > yv, 7 G 0(n), so that V n<m = 0(n)/0(n — 
m). The invariant measure dv on V n>m induced by the Lebesgue measure on the 
ambient space is defined up to a constant multiple. We normalize it using geometric 
argument and set 

(2.1) <T n ,m= / dV = Y[\S n -% 

V n , m 1=1 

where = 2tt^ +1 ^ 2 /T((i + l)/2) is the surface area of the i-dimensional unit 
sphere. 

Let n be the cone of positive definite symmetric matrices r = (r i j) mxm with 
the elementary volume dr — Yli<j^ r iJ- We denote |r| = det(r) and let d*r = 
| r |-(m+i)/2^ r ^ e GL(m, M)-invariant measure on Q. If T m is the group of 
upper triangular m x m matrices t — (Uj) with positive diagonal elements, then 
each r G f2 has a unique representation r = t't. 

We will constantly use the polar coordinates and the spherical coordinates on 
jgmxm w bich are defined as follows. 

Lemma 2.1. ([Hj3 pp. 66, 591], [Ma]) If x G R" xm , rank(x) = m, n > m, then 

(2.2) x ^vr 1/2 , v G V n . m , r = x'x G f2, 
and dx = 2- m \r\ { * n - m - 1 V 2 drdv. 

Lemma 2.2. ([F], E3|) If x G R" xm , rank(x) = m, n > m, then 
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dx = J^J tj j J dtj j dt^dv, dt* — JJ^ dtij. 

3 = 1 i<3 



The Schwartz space S = S(M. nxm ) is identified with the respective space on R" m 
of infinitely differentiable rapidly decreasing functions. The Fourier transform of a 
function / G L x (K" xm ) is defined by 



(2.3) 



(J7)(y) 



The relevant Parseval equality reads 

(2.4) (?f,r<p) = (2n) rm (f,<p), (/,¥>) 



f{x)tp(x) dx. 



2.2. The composite power function. Given r = (ri.j) € fi, let Ao(r) = 1, 
Ai(r) = ri i, A2(r), . . ., A m (r) = |r| be the corresponding principal minors which 
are strictly positive. For A = (Ai, . . . , A m ) G C m , the composite power function of 
the cone f2 is defined by 

Aj(r) 



(2.5) 



n 



A(_i(r) 



Ax(r)" 



.A m _x(r)" 



A TTI _ i -A r 



"A m (r)" 



In the special case Ai = . . . = A TO = A we write Ao = (A, . . . , A) (g C m ) so that 
r A = | r |A/2 If r = i / tj t = ^ £ thcn r A = j J™ ! / . This implies the 
following equalities: 

(2.6) r A+ " = r A r^, r A+a ° = r A |r|"/ 2 , a = (a, . . . , a); 

(2.7) (t'rt) x = (t't) x r x , t G T m . 

The reverses of A = (Ai, . . . , A m ) and r = (rjj) £ il are defined by 

" 1 



A* — (A m , . . . , Ai); 



so that 



(A*)j — A m — (^*)i,i — ^*m— i+l,m— i+1 ■ 



We have 
(2.8) 

The gamma function of the cone £1 is defined by 



/, 771 

(2.9) r n (A) = / r A e- tr W^r = Tr™^ 1 )/ 4 J[ T((X, - j + l)/2); 

see, e.g., jFKI p. 123]. The integral in (|2.9|l converges absolutely if and only if 
Re Xj > j — 1 for all j = 1, . . . , m, and extends meromorphically to all A G C m . 
The following relation holds: 



(2.10) 



J r x e-^ rs) d*r = T a (X) s~ x * , seQ 
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An important particular case of (|2.9|) is the Siegel integral 



~ tr(r) <L 



m(m— 1)/4 



f]r(A-j/2), ReX>(m-l)/2. 



3=0 



(2.11) r m (A) = / \r\; 

Q 

If A = (A, . . . , A), then Tn(Ao) = T m (A/2). The volume a n ^ m of the Stiefel mani- 
fold V n>m may be written in terms of the Siegel Gamma function: 

2m,^.nm/2 



(2.12) 



T m (n/2)- 



2.3. Radon transforms on the space of matrices. The main references for this 
subsection are |URT] . |UR1] . |UR4] . 0, [STT] . |Sh2| . We fix positive integers k,n, 
and m, < k < n, and let be the Stiefel manifold of orthonormal fc-frames in 
E™. For £ e K,fe and i 6 R fexm , the linear manifold 



(2.13) 



T = T(£,t) = {xeR nxm :£x = t} 



is called a matrix (n — k) -plane in M. nxm . We denote by T the set of all such planes. 
Each t £ H is an ordinary (n — /c)m-dimensional plane in W lm , but the set T 
has measure zero in the manifold of all such planes. The matrix Radon transform 
f(x) — > {TZkf){r) assigns to a function f(x) on R" xm a collection of integrals of / 
over all matrix planes t £ 1, namely 



(Kkf)(r) = / f(x) 
Precise meaning of this integral is the following: 
(2-14) (K*/)(r)= J fU 

where £ SO(n) is a rotation satisfying 
(2.15) = f, Co = 





4 



The following statement is a matrix generalization of the so-called projection- 
slice theorem. It links together the Fourier transform l|2.3|l and the Radon transform 
(|2.14fl . In the case m = 1, this theorem can be found in [Na, p. 11] (for fc = 1) and 
[Ke, p. 283] (for any < fc < n). 

Theorem 2.3. C |UR4| ) For f £ L 1 (W lXm ) and 1 < m < k, 

(2.16) (Ff)(tib) = [HKkfm •)](&), t£V n ,k, b £ R kxrn . 

Here, J-tp denotes the Fourier transform of a function t — > i) on the space 



3. Main results 
For A = (Ai, . . . , A m ) £ C", we introduce intertwining operators 

(3.1) (T x f)(u)= J f(v)(u'vv'u) x dv, u£V ntm , n>m, 

v„, m 
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that commute with the left action of 0(n). We call T x f the composite cosine 
transform of f. If Ai = . . . = X m = A, then (|3.1|1 reads 

(3.2) (T X f)(u)= J f(v)\det(v'u)\ x dv. 

v„ >m 

If / is a 0(m) right-invariant function on V n . m , then (|3.2|) can be identified with 
(|1.5|) (for I = to) and represents the usual A-cosine transform on G n ,m- 

Definition 3.1. We denote by £ the set of all A = (Ai, . . . , A m ) 6 C m satisfying 
Re Xj > j — in — 1 for all j = 1 , . . . , m. 

This definition is motivated by the following. 

Theorem 3.2. For f G L 1 (I^i,m), the integral (T x f)(u) converges absolutely a.e. 
on V niTn if and only if A G Z, and represents an analytic function of X in this 
domain. For such X, the linear operator T x is bounded on L 1 (V^ >m ). 

This statement follows immediately by Fubini's theorem from the equality [OR2 



[,, i \X j 2 m 7r m "/2 ro(A + m ) A 
(3-3) / (u w u) du = — -jt——, (= T 1 

J r m (TO/2j ro(A + n ) 

v n , m 

which is of independent interest. 

It is challenging to describe the set of all A G C m for which T x is injective. We 
cannot solve this problem in full generality and restrict our consideration to the 
space L (Vn,m) of 0(m) right-invariant integrable functions on V n , m . This allows us 
to obtain a precise description of those A for which T A is injective in the following 
important cases (a) 2to < n, A = (Ai, . . . , A m ) G C m , and (b) Ai = • • • = A m = A G 
C, provided that T x f and T x f exist as absolutely convergent integrals. Note that 
for to = 1, the space £ (V^, m ) is actually the space of even integrable functions on 
the unit sphere. 

Theorem 3.3. Let n > m > 1 and X G £, i.e. Re Xj > j — m — 1 for all 

j = 1, . . . , to. If, moreover, 

(3.4) Xj + m-j ^0,2,4, ... for all j = l,...,m, 

then the composite cosine transform T x is injective on L (V n _ m ). If 2m < n 



ana 



d \3-4\) fails, then T x is non- injective. Specifically, it annihilates all 0{m) 
right- invariant, harmonic, determinantally homogeneous polynomials of degree k > 
max{Aj + m — j} (see Definition \5.1)) . 



3 



Some comments are in order. The essence of our approach is that we apply the 
standard Fourier transform technique to obtain a higher-rank analog of i|1.3|) . We 
do not know if the condition l|3.4|l is necessary for injectivity of T x when 2m > n. 
To answer this question, one has to treat T x on polynomial representations of 
SO(n). These are parameterized by highest weights (mi,TO2, ■ ■ ■ , TOr n /2]) that are 
more general than those adopted in the framework of our approach; cf . |Strlj , jTTj . 
However, if Ai = ■ • • = A m = A, then for the A-cosine transform 1)1. 5[l . we give the 
following complete answer which reveals essential difference between the rank-one 
case and that of a higher-rank. 
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Theorem 3.4. Let n > m, Re X > — 1, and let r n , m — rank(G n . m ) = min(m, n—m) 
be the rank of the Grassmannian G n . m . If r n>m = 1, then T A is injective on 
L 1 (G„ jTO ) if and only if X 0,2,4,.... If r n>m > 1, i/ien T A is injective on 
i 1 (G„ jTO ) i/ and on/y if A 7^ 0, 1, 2, ... . 

This statement is known. It follows from the more general result of Alesker pQ . 



4. The generalized zeta integrals and the composite cosine 

transforms 

4.1. The generalized zeta integrals. In accordance with the polar decomposi- 
tion x = vr 1 ' 2 , v S Ki, m , r — x'x G f2, we introduce the generalized zeta integrals 
(or zeta distributions): 



(4.1) Z(4>,\,f) = / r x f(v) <f>(x) dx = (r A / 



(4.2) Z*(<f>,\,f) = J rtf(v)<j>(x)dx = (rlf,cj>), 

where / £ L l (V n!m ) and <j> G S'(R nxm ). Zeta integrals of this type with the angle 
component / = 1 are well known and arise in different occurrences; see |FK| . BSZ ) 
and references therein. We denote 

(4.3) A = {A e C m :ReXj>j-n-l for all j = 1, . . . , m}, 

(4.4) A = {A G C m : Xj = j — n — l for some 

j e {1, ...,m}, and J e {1,3,5,...}}. 

Lemma 4.1. The integrals \4-l\l and are absolutely convergent if and only 

if A G A, and extend as meromorphic functions of A with the polar set Aq. The 
normalized zeta integrals 



£*(0,A,/) 



(4.5) Z<U A, /) = Z°M A, /) = , 

rs;j(A + no) r^(A + n ) 

no = (n, . . . , n), are entire functions of A. 

To prove this lemma it suffices to write both integrals in spherical coordinates 
and then apply a standard argument from |GShj : see |OR2| for details. 

4.2. The basic functional equation. The connection between zeta integrals and 
composite cosine transforms can be established in the form of a functional equation 
which is actually the usual Parseval equality. Note that the function (T x f)(u), 
initially denned for u G V n ,mi extends to all matrices y G R nxm of rank m. Indeed, 
by H2.7J1 . in spherical coordinates y — ut, u G V n . m , t G T m , we have 

(4.6) (T x f)(y)=r x (T x f)(u) 
where r x = {t't) x = (y'y) x is the "radial part" of (T x f)(y). 
Theorem 4.2. Let 

(4.7) <p x (x) = X ' "°. /(«), x = vr x '\ n = (n,...,n), A G C m . 

t n(—A*) 
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If f is an integrable 0(m) right- invariant function on V n . m , then 

(4.8) (f<p x )(y) = - ^ X A T X f)(y), c x = 2^W m2 ' 2 /a m .^ 

Ta(X + m ) 

in the sense of S' -distributions. In other words, for each <f> G S(W nxm ), 

^ FTYT V ^ Ft) = (2n) nm (<p x , <j>) ee (27r)™" Z°(<f>, A* - n 0) /). 

A self-contained proof of this statement is given in OR2 . Here, we apply an 
alternative approach which is of independent interest. The main idea is to reduce 
the problem to the corresponding functional equation containing zeta integrals on 
l^mxm w jt n the angle component / = 1. To this end, we invoke the higher-rank 
Radon transform Ij2.14|l . We start with two auxiliary lemmas. 

Lemma 4.3. For r G il and y G R nxm , 

r r -A„-n , o-|A| nm/2 

Lr n (-A»)J r o (A + n ) 

in the sense of S' -distributions. 

Proof. Formula (|4.1Q(I was established by Khekalo jKhj who modified the argument 
from [St2, Chapter III, Sec. 3.4] for functions of matrix argument; see also |FK| . 
OR4 , |Ru4j on this subject. For convenience of the reader, we outline the proof 
of (|4.10() in our notation. Since 

F[e- tl( - xsx '^](y) = (2Tr) nm \s\^ n/2 e~- tI ^ ys ~ ly '\ 
for s G fl and <f> G S(R nxm ), the Parseval equality yields 

(4.11) | S |- n / 2 / e - tr (*!/«-V) (T(j)){y) dy= f e - tl( - xsx ' /4 ^0(xj dx. 



We multiply l|4.11|l by s A+n ° and integrate against d*s. This gives 



h (y) (F<t>) (y) dy= J I 2 (x) (j)(x) dx, 
where 

h(y) = J s x e~ tr ^ ys ' lyl) d,s, h{x) = J s x+no e- t,: ^ sx '/^d*s. 
n si 
Evaluation of the last integrals by means of (|2.1U|) gives 

h{y) = ^ 2 Y n (-\*){y'y) X , ReX 3 < j - m, 

and 

I 2 ( x ) = (47r)(l A l+" m )/ 2 r (A + n )(x'x)- x '- n °, Re \j >j-n-l. 
Hence, if j — n — 1 < Re \j < j — m, then 

(4.12)r n (-A*) / (y'y) x (^)(y)dy^c x Tn(X + n Q ) f (x' x)- x *- n °0(xj dx, 



where c x = 2 ,lm +l A l7r™" 1 / 2 . By Lemma f4. II this extends analytically to all A, and 
we are done. □ 
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Lemma 4.4. For cf) E S(R nxm ) and £ e V n , m , 

R" Xm Rmxm 

(4.14) d\ = 2 mn+ \ x \-K nm ~ m2 / 2 . 

Proof. Let us denote by A(£) the left side of (|4.13|) . By passing to the spherical 
coordinates on R nxm , according to Lemma \2. 21 and (|2.7|) we obtain 

m = ro(A + mo ) J II*ii +n ~ , *^ di * y (^X^KeeV^- 

By Theorem ^. 21 this integral converges absolutely if A G £, that is i?e Xj > j — to— 1 
for all j = 1, ... , to. Let us evaluate for such A. 



We put x = g^ 



t 



, where £ SO(n) is a rotation satisfying (|2.15|l with 



k = m,uje M("- m ) xm , and i £ R m ><"\ Then fa; = t, so that the Fubini theorem 
and l|4 lOf) yield 



1 


r n (A + 


m ) 

K 


i 




r n (A + 


mo) 


2| A|+m 2 





^(0 = iwTT^ / (*'*) Arf * / (^)(% 



u> 
t 



)du> 



{t't) x {n m T(t>){i,t) dt 



r n (-A*) 

2 |A| 7r -™ 2 /2 

Tn(— A*) 



(i't)- A *- mo (jr-i7^^)(e,t) dt 



(t't) t - A '- m »(« m ^)K,t)rft. 



Here, and f 1 denote the Fourier transform acting in the t- variable on the space 
R mxm , and its inverse, respectively. We have used the equality (^ r_1 (^)(^, t) = 
(27r)" m {T<p){Z,-t). Moreover, we have applied (|4.10|> with n — to in the case 
when both functionals in this equality are regular. The latter is true if 

(4.15) j — to — 1 < Re Xj < j — to. 

By (151711) . 

T[K m T^,-)]{t) = {TTmt) = (27r)™>(-^). 

This gives 

o I A I + n m —rim — 77i 2 /2 r 

A ^ = — t^k) — j ^ t )* x *~ mo m)dt. 

Therefore, 14.13|l is valid when A satisfies (|4.15J) . By Lemma 14.11 this extends 
analytically to all A G C m . □ 
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Proof of Theorem l4.21 Let A obey lj4.15|L so that by Theorem ^ . 21 and Lemma 
14.11 integrals in (|4.9|l are absolutely convergent. Then, by (|4.13(l and the Fubini 
theorem, 

1 1 



{T f, J~(f>) = ^^——^ {T*f){x){F4>){x)dx 



T n {\ + m Q y ^ r o (A + m 

1 



r n (A + m ) 

v, 



f(v)dv I {Fcj))(x)(x'vv'x) x dx 



r n (-A*) 



f{v)dv / {t't)- x '- mo 4>{vt)dt 



Here, d\ is defined by (|4.14|l . By the polar decomposition (see Lemma |2~T|) . for the 
right-invariant function / we obtain 



_ i _ (TVi ^ _ 2_^_ j [{v)iv j dj j r: ^ ]rr ^ {virl/2)dr 

V n , m O(m) Q 

2- m d x a„ 



v„ m a 



f(v)dv / r; A *- m °|rr 1/2 0(w 1 /2)rf r 



d\a 

r n (-A,) 



f{x{x'x)- l l 2 ){x'x)- x *- tl0 ^{x)dx. 



Hence, (|4.9f) is proved for j — m — 1 < Re Xj < j — m. According to Lemma f4. II 
this extends analytically to all A £ C m . 

Example 4.5. Let to > 1, Ai = . . . = X m = A, \x\ m = det(x'x) 1 ^ 2 . Then 

= rlf-X/2) fW x ' x )~ 1/2 )> ( TX f)(y)= J f(v)\det(v'y)\ x dv. 

v n>m 

If / is an integrable 0(m) right-invariant function on V n , m , then the operator T x 
coincides with (|1.5I) . and we have 

o — Am 7r m 2 /2 

(4.16) (?<Px)(y) = r 77Y- W9 , (r A /)(y). 

cr m , m l m ((A + m)/2) 
If m = 1, and x, y eR n \ {0}, then 

^(x) = ^/(^), (TV)(,)= / /(«)|« -y|^. 

In this case, 

(4.17) ^)^ = f9rrp) (TA/)(y) - 

The last equality is well known and can be found in many sources. 
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4.3. The case of homogeneous polynomials. Let Pk(x) be a polynomial on 
l" xm which is harmonic (as a function on M. nm ) and determinantally homogeneous 
of degree fc, i.e., Pk(xg) = det(g) k Pk(x), Mg £ GL(m,M). It means that Pk is a 
usual homogeneous harmonic polynomial of degree km on M. nm . Theorem 14.21 can 
be strengthened if we choose / to be the restriction of Pk(x) onto V n ^ m . 



Theorem 4.6. Let 



r — A» — no 



(4.18) tp x , k (x) = * - r Pk(v), x^vr 1 ' 2 , d x = 2-Mn nm /H km . 

J- n(— A*J 

Then, for all A e C m , 

(4.19) (^A, fc )(«r 1/a ) = r r l^yiSlMn^ ^^ 

rn(-A*jrn(A + ko + n ) 

in i/ie sense of S' -distributions. In other words, for each (j) G S^R™*™), 
4rfi(k °" A,) ^Wr\^) = (2,r)™W, 



r o (A + k + n ) 

= (27r)" m ZO(0,-A, -n ,P fe ). 

Proof. The classical Hecke identity 

/ ft(l)e ^.)^.) i( = ^ )e ^) 

implies that 

(4.20) | S r fc -"/2 f P^e-^y-W (F<l>)(y)dy 



(2jTi)- km / P fe (a;)e- tr ( xsa: '/ 47r ^(a;)da;. 



We multiply this by s A + n o+ko anc j p roce ed, as in the proof of Lemma PDA to obtain 
the result for j — n — k — 1 < Re Xj < j + k — m. Since this domain is not void for 
all k = 0, 1, 2, . . ., and the normalized zeta integral Z®(<fi, — A, — no, Pk) is an entire 
function of A, the result follows by analytic continuation. □ 

Example 4.7. Let m > 1, Ai = . . . = X m = A, |a;| m = det^'x) 1 / 2 , 

I I— A — 71 

VX,k(x) = J T " ... Pkixix'x)- 1 ' 2 ), d x = 2 -^V m / 2 t km . 

t ml. - 'V-'J 

Then 

(4.21) (J> A , fc )(tf) = ^^Zgg | y ^P fe ( y ( y ' y )-^). 

L m (-A/2)r m ((A + fc+n)/2) 

If m = 1, then ipx,k(x) = r(-\/2) Pk (yfy) , and we have 

< i22 > '^)M- ix-^mt>" + ^)/») W >ft (w)- 
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5. Proofs of main results 
5.1. A higher-rank analog of (|1.3|> . 

Definition 5.1. |Herz| A polynomial Pk(x) on M. nxm is called an H -polynomial of 
degree k if it is 0(m) right-invariant, harmonic, and determinantally homogeneous 
of degree k. We denote by Ttk the space of all such polynomials. 

Lemma 5.2. Let Pk £ Tlk, 

(5.D MA) = p r f ■ + ■ m r f x f° ~ y xscr. 

r«(-A*) rn(A + k + n ) 
If A does not belong to the polar set ofTn(X + mo), then 

(5.2) (T X P k )(vr 1 / 2 )=c» k (X)P k (v)r X , c = Tr m ( n ~ m V 2 i km a m , m , 
in the sense of S' -distributions. 

Proof. Let us compare l|4.8|) and l|4.19|l . assuming f(v) — Pk{v). For all A £ C m , 



in the sense of ^'-distributions. If we exclude all A belonging to the polar set of 
rn(A + m ), we get (S2J|. □ 

Corollary 5.3. (Cf. Q) If X £ 2 (see Definition]^ and P k £ H k , then 
(5.4) (T X P k )(v) = c^k(X) P k (v), v £ V n , m , 

c and fJ-k(X) being the same as in f5.ty) . 

Proof. The function To (A + mo) has no poles in £. Hence, by l|5.2[) . we have 
((T x P k )(vr 1 / 2 ),(f)) = c^ fc (A)(P fe (v)r A ,0) for all <j>(y) = ^vr 1 ' 2 ) £ S(M. nxm ). 
Choose 4>(y) = x( r ) V , ( w )? where x( r ) is a non-negative C°° cut-off function sup- 
ported away from the boundary of £1 and ip{v) is a C°° function on V n _ rn . By 
passing to polar coordinates, owing to l|4.6|l we obtain 



[{T x P k )(v) - cfi k (X)P k {v)}^(v)dv = 0, c x = const ^ 0. 
This implies (|5.4|l . □ 



Remark 5.4. An important question is, do there exist ff-polynomials of a given 
degree k? For n — m we have exactly two such polynomials, namely, Pq[x) = 1 
and Pi(x) = det(aj). It is known |Herzl p. 484] that for 2m < n there exist H- 
polynomials of every degree k. The space Hk in this case is spanned by polynomials 
of the form Pk(x) = det(a'a;) fc where a is a complex nxm matrix satisfying a' a = 
|TTl p. 27]. 
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5.2. Proof of Theorem l3.31 To prove the first statement, we consider the equality 



m 



(5.5) ((T x f)(vr^ 2 ), F<t>) = A{\) (r," A *- n ° f(v), 0), 

A(X) = (2ir) nm T n (\ + ma )/c x Tn(-K), 

which follows from (|4.9|) and (|4.7|) . Suppose that (T x f)(v) = a.e. on V n . m for 
some A G £. Then (T A /)(y), y = vr 1 ' 2 G W ixm , is zero for almost all y G tt" xm , 
and l(53|) yields A(A) (r\,r A *~ no /(«), 0) = 0. The assumption (|3~%j) along with 
A G £ imply i?e Aj > j — m — 1 and Aj ^ j — m, j — in + 2, . . . . Hence A is not a 
pole of Tq(— A*), and therefore ^4(A) ^ 0. This gives (r~ x *~ n ° f(v), (j)) — 0, where 
the left side is understood in the sense of analytic continuation. Choosing <p as in 
the proof of Corollary 15. 31 we obtain f(v) = a.e. on V n , m . 

To prove the second statement, we note that, for 2m < n, _ff-polynomials of 
every degree k do exist. We observe that the function ro(ko — A») in /Xfc(A) can be 
written as 

r n (k - a,) = tt"^- w f[ r [ k + J -^- 

3=1 

It has no poles in £ if k > max{Xj + m — j}. Since Tq(X + m ) also has no poles 

j 

in £, then, by 15.4fl . T X P^ = for all such k provided that A is a pole of Fq(— A*) 
(i.e., i|3.4[l fails). This proves the theorem. 

5.3. Proof of Theorem 13.41 We make use of the canonical homeomorphism of 
G n ,m and G n ,n-m under which T x is invariant. To be precise, given u, v G Vn.m-, let 
S, G G n ,m and n G G n .m be m-dimensional subspaces of R™ spanned u and v, respec- 
tively. We denote by £ G G n ,n-m and ?7 G G„ jtt _ m the corresponding orthogonal 
subspaces, and choose any u± G V n ,n-m in £ x an d v ± S Ki,n-m in r/- 1 . Since 
= [ ? ?" L |^" L ]; we can successively define the functions F(n) on G n ra , F^{ri^) on 
G„,„_ m , and on 7„, n - m by F(r?) = /(«), F ± (^) = ^XV)^), = 

F±(V' L )- Then T A expresses through the similar operator T A on V n , n -m as follows: 

(5-6) (T A /)(u) = (T A / ± )K). 

Indeed, 

(T X f)(u) = | F(r,)[ ?7 |C] A dr^ | [m^Y V 

G„, m G„,„_ m 

/ ± K)|det« U± )| A ^ ± - (T A / ± )( U± ). 



If rank(G„. m ) = 1, i.e., m — 1 or n — 1, the result follows from the multiplier 
equality (|1.3|) which is a particular case of (|5.4|) . 

Let rank(G„ !m ) > 1. Owing to l|5.6|l . it suffices to prove the theorem for 2m < n, 
because otherwise we have 2(n — m) < n, and one can treat T A instead of T A . If 
Ai = . . . = A m = A, the condition A 7^ 0,1,2,... coincides with (|3.4|) . But for 
2m < n, the result follows from Theorem 13. 31 
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